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The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )K Responsive to communication(s) filed on 24 January 2002 . 
2a)D This action is FINAL. 2b)03 This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quay/e, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-22 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) £3 Claim(s) 7-22 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) Q The specification is objected to by the Examiner. 

10)K) The drawing(s) filed on 24 January 2002 is/are: a)[X] accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
1 1 )□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-1 52. 

Priority under 35 U.S.C. § 119 

12)D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)D All b)D Some * c)Q None of: 

1 .□ Certified copies of the priority documents have been received. 

2.Q Certified copies of the priority documents have been received in Application No. . 

30 Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 

1. This action is responsive to communications: application filed on 1/24/2002, IDS filed 
on 1/24/2002. 

2. Claims 1-22 are pending in the case. Claims 1,16, and 22 are independent claims. 

Claim Rejections - 35 USC § 102 

3. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 1 22(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 351(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

4. Claims 1-22 are rejected under 35 U.S.C. 102(e) as being anticipated by Stubler et 
al. (hereinafter "Stubler"), US 6,804,684 B2 filed 5/7/2001. 

Regarding independent claim 1, 16, and 22, Stubler discloses actively selecting 
examples of multimedia content to be annotated by a user and accepting input annotations from 
the user for the selected examples in fig. 2, fig. 6-7, col. 3 line 46 - col. 4 line 12, col. 8 lines 18- 
23, and col. 9 line 65 - col. 10 line 18. Stubler discloses propagating the input annotations to 
other instances of multimedia content and storing the input annotations and the propagated 
annotations in fig. 2, col 3 line 46 - col. 4 line 12, and col. 8 lines 18-55. 

Regarding dependent claims 2 and 17, Stubler discloses wherein the step of actively 
selecting is performed using a selection technique from the group consisting of deterministic and 
probabilistic in col. 4 line 64 - col. 5 line 19. 
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Regarding dependent claims 3 and 18, Stubler discloses wherein the step of actively 
selecting which is performed deterministically or probabilistically, is based on explicit models 
and feature proximity/similarity measures, and returns one or more examples of multimedia 
content to be annotated in fig. 2 and col. 8 lines 18-55. 

Regarding dependent claims 4 and 19, Stubler discloses wherein the step of actively 
selecting, which is performed deterministically or probabilistically, is based on implicit models 
and feature proximity/similarity measures, and returns one or more examples of multimedia 
content to be annotated in fig. 2 and col. 8 lines 18-55. 

Regarding dependent claims 5 and 20, Stubler discloses wherein an optimization 
criterion for active selection includes one or more criteria selected form the group consisting of: 
maximizing disambiguation, information measures, and confidence in fig. 2 and col. 8 lines 18- 
55. 

Regarding dependent claims 6 and 21, Stubler discloses wherein the multimedia 
content comprises one or more types selected from the group consisting of: images, audio, video, 
graphics, text, multimedia, Web pages, time series data, surveillance data, sensor data, relational 
data, and XML data in col. 3 line 46 - col. 4 line 12. 

Regarding dependent claim 7, Stubler discloses wherein the input annotations are 
created by a user with reference to a vocabulary in col. 3 line 46 - col. 4 line 12 and col. 8 lines 
18-23. 

Regarding dependent claim 8, Stubler discloses wherein the vocabulary contains one or 
more items selected from the group consisting of: terms, concepts, labels, and annotations in col. 
3 line 46 - col. 4 line 12 and col. 8 lines 18-23. 
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Regarding dependent claim 9, Stubler discloses wherein the process of creating input 
annotations by the user involves multimodal interaction with the user using graphical, textual, 
and/or speech interface in fig. 2 and col. 3 line 46 - col. 4 line 12. 

Regarding dependent claim 10, Stubler discloses wherein the input annotations are 
created by means of steps selected from the group consisting of: creating new annotations, 
deleting existing annotations, rejects proposed annotations, and modifying annotations in fig. 6-7 
and col. 9 line 65 - col. 10 line 18. 

Regarding dependent claim 11, Stubler discloses wherein the vocabulary is adaptively 
or dynamically organized and/or limited by the system of the user in fig. 6-7 and col. 9 line 65 - 
col. 10 line 18. 

Regarding dependent claim 12, Stubler discloses wherein the multimodal interaction 
involves speech recognition, gaze detection, finger pointing, expression detection, and/or 
effective computing methods for sensing a user's state in fig. 6-7 and col. 9 line 65 - col. 10 line 
18. 

Regarding dependent claim 13, Stubler discloses wherein the determination of the 
propagation of annotations is made deterministically or probabilistically and on the use of 
models for each annotation or for joint annotations in col. 4 line 64 - col. 5 line 19. 

Regarding dependent claim 14, Stubler discloses wherein the models are created or 
learned automatically or semi-automatically and/or are updated adaptively from interaction with 
the user in fig. 2, col. 3 line 46 - col. 4 line 12, and col. 8 lines 18-55. 
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Regarding dependent claim 15, Stubler discloses wherein the models are based on 
nearest neighbor voting or variants, parametric or statistical models, expert systems, rule-based 
systems, or hybrid techniques in fig. 2, col. 3 line 46 - col. 4 line 12, and col. 8 lines 18-55. 

Conclusion 

5. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. Zhang et al., US 6,748,398 B2 filed 3/30/2001 discloses relevance-feedback, 
content-based facilitating accurate and efficient image retrieval minimizing the number of 
iterations of user feedback regarding the semantic relevance of exemplary images while 
maximizing the resulting relevance of each iteration. Lipson et al., US 6,549,660 Bl 
continuation filed 2/12/1996 discloses identifying and classifying images. Li et al., US 
6,094,653 filed 12/29/1997 discloses classifying words into word clusters to probabilistically 
indicate whether a document is in a particular category. Golshani et al., US 6,594,386 Bl filed 
4/22/1999 discloses computerized indexing and retrieval of digital images. Chang et al., US 
6,741,655 Bl provisional filed 5/5/1997 discloses object-oriented content-based video search. 
Luo et al., US 6,826,316 provisional filed 1/24/2001 discloses determining image similarity. 

6. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Peter J Smith whose telephone number is 571-272-4101. The 
examiner can normally be reached on Mondays-Fridays 7:00am-3:30pm. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Joseph H Feild can be reached on 571-272-4090. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



PJS 

12/17/2004 




